Imaging device for distinguishing foreground

ABSTRACT

An imaging device including a condenser lens and an image sensor is provided. The image sensor is configured to sense light penetrating the condenser lens and includes a pixel matrix, an opaque layer, a plurality of microlenses and an infrared filter layer. The pixel matrix includes a plurality of infrared pixels, a plurality of first pixels and a plurality of second pixels. The opaque layer covers upon a first region of the first pixels and a second region of the second pixels, wherein the first region and the second region are mirror-symmetrically arranged in a first direction. The plurality of microlenses is arranged upon the pixel matrix. The infrared filter layer covers upon the infrared pixels.

RELATED APPLICATIONS

The present application is a continuation application of U.S.application Ser. No. 16/101,942, filed on Aug. 13, 2018, which is acontinuation application of U.S. application Ser. No. 15/189,189, filedon Jun. 22, 2016, which is based on and claims priority to TaiwaneseApplication Number 104129150, filed Sep. 2, 2015, the disclosure ofwhich is hereby incorporated by reference herein in its entirety.

BACKGROUND 1. Field of the Disclosure

This disclosure generally relates to an optical image sensor and, moreparticularly, to an imaging device capable of distinguishing foregroundand background and an operating method thereof.

2. Description of the Related Art

Activating a predetermined application of a system by recognizinggestures has been broadly applied to various electronic devices, e.g.,portable electronic devices or wearable electronic devices. Aconventional method for distinguishing foreground and background adoptsa system light source to illuminate the object located within anoperating range. When the system light source is illuminating light, animage sensor captures a bright image; whereas, when the system lightsource does not illuminate light, the image sensor captures a darkimage. By calculating a differential image between the bright image andthe dark image, it is able to cancel the interference from background.

However, the portable electronic device and the wearable electronicdevice are generally operated under strong light, e.g., under sunlight.As the brightness of the system light source is very weak compared tosunlight, the background interference can not be effectively cancelledby only calculating the differential image between bright and darkimages.

Accordingly, an imaging device without being limited by ambient lightintensity is required in the art.

SUMMARY

The present disclosure provides an imaging device capable ofdistinguishing foreground and an operating method thereof thatincorporate the phase detection to overcome the limitation caused bystrong ambient light.

The present disclosure further provides an imaging device capable ofdistinguishing foreground and an operating method thereof that operatein different operating modes under different ambient light intensity.

The present disclosure provides an imaging device including a condenserlens and an image sensor. The image sensor is configured to sense lightpenetrating the condenser lens and includes a pixel matrix, an opaquelayer, a plurality of microlenses and an infrared filter layer. Thepixel matrix includes a plurality of infrared pixels, a plurality offirst pixels and a plurality of second pixels. The opaque layer coversupon a first region, which is a part of each first pixel, and upon asecond region, which is a part of each second pixel, wherein the firstregion and the second region are symmetrically arranged in a firstdirection, and uncovered regions of the first pixels and the secondpixels are arranged to be larger at a pixel edge than at a pixel center.The infrared filter layer covers upon the infrared pixels.

The present disclosure further provides an imaging device including acondenser lens and an image sensor. The image sensor is configured tosense light penetrating the condenser lens and includes a pixel matrix,an opaque layer, a plurality of microlenses and an infrared filterlayer. The pixel matrix includes a plurality of infrared pixels, as wellas a plurality of first pixels and second pixels arranged at a part ofpixel columns of the pixel matrix. The opaque layer covers upon a firstregion, which is smaller than an area of the covered first pixel, ofeach first pixel, and upon a second region, which is smaller than anarea of the covered second pixel, of each second pixel, whereinuncovered regions of one first pixel and one second pixel aresymmetrically arranged and adjacent to each other. The infrared filterlayer covers upon the infrared pixels.

The present disclosure provides an imaging device including a condenserlens and an image sensor. The image sensor is configured to sense lightpenetrating the condenser lens and includes a pixel matrix, an opaquelayer, a plurality of microlenses and an infrared filter layer. Thepixel matrix includes a plurality of infrared pixels, a plurality offirst pixels and a plurality of second pixels arranged in a firstdirection and a second direction. The opaque layer covers upon a firstregion which is a part of the first pixels and upon a second regionwhich is a part of the second pixels, wherein the first region and thesecond region are symmetrically arranged in the first direction, and anuncovered region of one first pixel is adjacent to an uncovered regionof one second pixel in the first direction. The infrared filter layercovers upon the infrared pixels.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects, advantages, and novel features of the present disclosurewill become more apparent from the following detailed description whentaken in conjunction with the accompanying drawings.

FIG. 1 is a schematic block diagram of an imaging device according toone embodiment of the present disclosure.

FIGS. 2A-2B are cross sectional views of an image sensor of an imagingdevice according to some embodiments of the present disclosure.

FIGS. 3A-3D are schematic diagrams of the arrangement of an opaque layeraccording to some embodiments of the present disclosure.

FIG. 4 is a schematic diagram of an operating method of an imagingdevice according to one embodiment of the present disclosure.

FIG. 5 is a schematic diagram of an operating method of a first mode ofan imaging device according to one embodiment of the present disclosure.

FIG. 6 is a flow chart of an operating method of an imaging deviceaccording to one embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENT

It should be noted that, wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

Referring to FIGS. 1 and 2A-2B, FIG. 1 is a schematic block diagram ofan imaging device according to one embodiment of the present disclosure;and FIGS. 2A-2B are cross sectional views of an image sensor of animaging device according to some embodiments of the present disclosure.The imaging device 1 includes a condenser lens 10, an image sensor 11, aprocessor 13 and an infrared light source 15. In some embodiments, theprocessor 13 is disposed, for example, in a same chip together with theimage sensor 11 and the infrared light source 15. In some embodiments,the processor 13 is, for example, an external processing unit outsidethe image sensor 11, and is configured to receive and process imageframes F captured by the image sensor 11 so as to select a first mode(e.g., a normal mode) or a second mode (e.g., a strong light mode) toseparate at least one foreground image from the background image. Forexample, when the imaging device 1 is applied to the gesturerecognition, the at least one foreground image is, for example, a handof a user or an object held by a user. The processor 13 is, for example,a microcontroller (MCU), a central processing unit (CPU), a digitalsignal processor (DSP) or the like for processing image frames Foutputted by the image sensor 11.

The infrared light source 15 is, for example, a light emitting diode ora laser diode, and is configured to emit infrared light Ls to illuminatean operable range of the imaging device 1, wherein the operable range isdetermined, for example, by element parameters of the imaging device 1.When an object 9 enters the operable range, the object 9 reflects theinfrared light Ls toward the image sensor 11 to generate reflected lightbeams Lrs. In some embodiments, the imaging device 1 includes at leastone optical element (not shown) to homogenize the light emitted by theinfrared light source 15.

The condenser lens 10 is, for example, arranged in a lens of an imagecapturing device (e.g., a camera), and is a single lens or a lens setarranged along an optical axis without particular limitations. Forsimplification, a single lens is shown herein. The condenser lens 10 isserved as a lens window configured to condense the reflected light beamsLrs from the object 9 or ambient light La, and guide the reflected lightbeams Lrs and the ambient light La to the image sensor 11. A distancebetween the condenser lens 10 and the image sensor 11 is preferablyequal to a first focal length of the condenser lens 10 (e.g., the focallength close to a side of the image sensor 11). It is appreciated thatwhen the ambient light La exists, the reflected light beams Lrs alsocontains a part of ambient light reflected by the object 9.

The image sensor 11 (e.g., a pixel matrix being shown herein) senses,based on a predetermined focal length, light (e.g., reflected lightbeams Lrs and ambient light La) penetrating the condenser lens 10 andoutputs an image frame F. For example, the image sensor 11 outputs abright image frame when the infrared light source 15 is turned on, andoutputs a dark image frame when the infrared light source 15 is turnedoff. The image sensor 11 includes a pixel matrix (e.g., a 9×9 pixelmatrix being shown herein), an opaque layer 113, a plurality ofmicrolenses 115 and an infrared filter layer 117 (e.g., referring toFIGS. 2A and 2B), wherein the opaque layer 113 is patterned to cover atleast a part of a plurality of pixels included in the pixel matrix 111such that uncovered regions of the pixels receive incident light ofdifferent phases through different parts of the microlenses 115. Thepredetermined focal length is referred to a second focal length formedby the condenser lens 10 and the microlens 115 together as well as at alight incident side of the condenser lens 10. In the present disclosure,the predetermined focal length is sometimes referred to a predeterminedfocal length of the condenser lens 10 or the image sensor 11.

It is noticed that when an object 9 is at the second focal length (e.g.,the focal length away from the image sensor 11, i.e. the predeterminedfocal length) of the condenser lens 10 to reflect the infrared light Lsor ambient light La to the imaging device 1, the object image in theimage frame F outputted by the image sensor 11 does not have a positionoffset in the subframe corresponding to pixels under different coverpatterns; whereas, when the object 9 is not at the second focal length,the object image in the image frame F outputted by the image sensor 11has a position offset toward different directions in the subframecorresponding to pixels under different cover patterns (illustrated byexamples below). Accordingly, it is possible for the imaging device 1 toidentify whether the object 9 is a foreground image within the operablerange by identifying whether the position offset is within apredetermined range. In other words, the imaging device 1 of the presentdisclosure has an operable range, and the imaging in the image frame Fof an object within the operable range is referred to a foreground imageherein.

In one embodiment, the pixel matrix 111 includes a plurality of infraredpixels P_(inf), a plurality of first pixels P₁ and a plurality of secondpixels P₂ arranged along a first direction (e.g., X direction) and asecond direction (e.g., Y direction). It should be mentioned that in thepresent disclosure, the first pixels P₁ and the second pixels P₂ arereferred to pixels with different regions covered by the opaque layer113. For example, in a monochromatic image sensor, structures of thefirst pixels P₁ and the second pixels P₂ are identical, but only coverpatterns of the opaque layer 113 thereupon are different (as shown inFIG. 1). For example, in a color image sensor, the first pixels P₁ andthe second pixels P₂ respectively include red pixels (e.g., red filterlayer being formed thereon), green pixels (e.g., green filter layerbeing formed thereon), blue pixels (e.g., blue filter layer being formedthereon) or pixels of other colors, and cover patterns of the opaquelayer 113 upon the first pixels P₁ and the second pixels P₂ aredifferent. The infrared pixels P_(inf) are covered by an infrared filterlayer 117 but are not covered by the opaque layer 113.

The opaque layer 113 is, for example, formed by the metal layer servedas electrical paths (e.g., at least one layer of M1 to M10 in the CMOSprocess), a black light blocking layer formed in addition to said metallayer or a combination thereof without particular limitations as long asthe opaque layer 113 is able to block a part of incident light. In thisembodiment, the opaque layer 113 covers upon a first region (e.g., theregion filled with oblique lines) of the first pixels P₁ and upon asecond region (e.g., the region filled with oblique lines) of the secondpixels P₂. In FIG. 1, the first region is at a side of the pixel alongthe first direction (e.g., X direction), and the second region is at aside of the pixel along an inverse direction of the first direction. Thefirst region of the first pixels 1 and the second region of the secondpixels are mirror-symmetrically arranged in the first direction. Inaddition, The first pixels P₁ further include an uncovered region (e.g.,blank region) outside the first region, and the second pixels P₂ furtherinclude an uncovered region (e.g., blank region) outside the secondregion, wherein the uncovered region of the first pixels P₁ and theuncovered region of the second pixels P₂ receive incident light ofdifferent phases through different parts of the microlenses 115 (asshown in FIG. 2A).

For example in FIG. 1, the first region of the first pixels P₁ is at anupper side of the first pixels P₁, and the second region of the secondpixels P₂ is at a lower side of the second pixels P₂. It should bementioned that although FIG. 1 shows that the first region and thesecond region are substantially 50% of an area of a single pixel, it isonly intended to illustrate but not to limit the present disclosure. Inother embodiments, the first region and the second region are 5% to 95%of an area of a single pixel without particular limitations.

The microlenses 115 are disposed upon the pixel matrix 111, andrespectively aligned with one pixel. The opaque layer 113 and theinfrared filter layer 117 are sandwiched between the pixel matrix 111and the microlenses 115, wherein distances between the opaque layer 113and the pixel matrix 111 and between the infrared filter layer 117 andthe pixel matrix 111 may or may not be identical without particularlimitations. Accordingly, the first pixels P₁ and the second pixels P₂receive incident light of different phases respectively through a firstpart of the microlenses 115 (e.g., the lower part of the microlenses 115in FIG. 1 and the right part of the microlenses 115 in FIG. 2A) and asecond part of the microlenses 115 (e.g., the upper part of themicrolenses 115 in FIG. 1 and the left part of the microlenses 115 inFIG. 2A). It should be mentioned that although FIG. 2A shows thatuncovered regions of the first pixels P₁ and the second pixels P₂ aresubstantially a half of the microlenses 115, it is only intended toillustrate but not to limit the present disclosure. It is appreciatedthat light capable of penetrating the microlenses 115 to reach theuncovered regions is determined by the cover pattern of the opaque layer113. In the present disclosure, the first part and the second part ofthe microlenses 115 are arranged as 5% to 95% of the microlenses 115without particular limitations.

In FIG. 1, the pixel matrix 111 further includes a plurality of thirdpixels P₃ and a plurality of fourth pixels P₄ arranged along the seconddirection (e.g., Y direction). The opaque layer 113 further covers upona third region (e.g., the region filled with oblique lines) of the thirdpixels P₃ and upon a fourth region (e.g., the region filled with obliquelines) of the fourth pixels P₄, wherein the third region is at a side ofthe pixel along the second direction (e.g., Y direction), and the fourthregion is at a side of the pixel along an inverse direction of thesecond direction. For example in FIG. 1, the third region is at the leftside of the third pixels P₃, and the fourth region is at the right sideof the fourth pixels P₄, and the third region and the fourth region aremirror-symmetrically arranged in the second direction.

More specifically, the opaque layer 113 covers upon the pixel matrix111, and includes a first cover pattern covering upon a first region ofthe first pixels P₁, a second cover pattern covering upon a secondregion of the second pixels P₂, a third cover pattern covering upon athird region of the third pixels P₃, and a fourth cover pattern coveringupon a fourth region of the fourth pixels P₄, wherein the first regionand the second region are mirror-symmetrically arranged in a firstdirection, and the third region and the fourth region aremirror-symmetrically arranged in a second direction. In one embodiment,the first direction is perpendicular to the second direction. It shouldbe mentioned that the arrangement of the first pixels P₁ to the fourthpixels P₄ is not limited to that shown in FIG. 1, and preferable thefirst pixels P₁ to the fourth pixels P₄ are uniformly arrangedeverywhere in the pixel matrix 111. In addition, in some embodiments thepixel matrix 111 includes only the first pixels P₁ and the second pixelsP₂, or includes only the third pixels P₃ and the fourth pixels P₄according to different applications.

In one embodiment, all of the first region, the second region, the thirdregion and the fourth region of the pixel matrix 111 have identicalareas (as shown in FIG. 1), e.g., 5% to 95% of an area of a singlepixel.

It should be mentioned that although FIG. 1 shows that the first regionand the second region are rectangles mirror symmetrical in a firstdirection, and the third region and the fourth region are rectanglesmirror symmetrical in a second direction, it is only intended toillustrate but not to limit the present disclosure. In otherembodiments, the first region to the fourth region are not rectangles.For example referring to FIGS. 3A to 3D, the uncovered regions (e.g.,the blank region) of the first region to the fourth region are arrangedto increase or monotonically increase along a predetermined direction(e.g., a direction in which two pixels adjacent to each other), andshapes thereof are mirror-symmetrical in the predetermined direction.When the first pixels P₁ (or the third pixels P₃) and the second pixelsP₂ (or the fourth pixels P₄) are receiving incident light, as the phasedifference between the received incident lights close to the center ofthe first pixels P₁ (or the third pixels P₃) and close to the center ofthe second pixels P₂ (or the fourth pixels P₄) is not obvious, whereasthe phase difference between the received incident lights close to theedge of the first pixels P₁ (or the third pixels P₃) and close to theedge of the second pixels P₂ (or the fourth pixels P₄) is more obvious,the uncovered region close to the pixel edge is arranged to be largerthan the uncovered region close to the pixel center so as to improve theaccuracy of the phase detection. It should be mentioned that FIGS. 3A-3Dare only intended to illustrate but not to limit the present disclosure.

The processor 13 is configured to form an infrared subframe according toinfrared image regions I_(inf) in the bright image frame and the darkimage frame corresponding to the infrared pixels P_(inf) form a firstsubframe F_(P1) according to first image regions I_(P1) in the brightimage frame and the dark image frame corresponding to the first pixelsP₁ and form a second subframe F_(P2) according to second image regionsI_(P2) in the bright image frame and the dark image frame correspondingto the second pixels P₂, as shown in FIG. 4. The processor 13 alsocalculates, in a first mode, a differential image between the infraredsubframe F_(inf_B) of the bright image frame and the infrared subframeF_(inf_D) of the dark image frame to separate at least one foregroundimage, or separates, in a second mode, at least one foreground imageaccording to the first subframe F_(P1) and the second subframe F_(P2),wherein the first mode is, for example, a normal mode and the secondmode is, for example, a strong light mode. More specifically, the firstsubframe F_(P1) is formed by gray level data outputted by the firstpixels P₁, the second subframe F_(P2) is formed by gray level dataoutputted by the second pixels P₂, and the infrared subframe F_(inf) isformed by gray level data outputted by the infrared pixels P_(inf).

When the pixel matrix 111 includes the arrangement of four pixel types,the processor 13 is further configured to form a third subframe F_(P3)according to third image regions I_(P3) in the bright image frame andthe dark image frame corresponding to the third pixels P₃ and form afourth subframe F_(P4) according to fourth image regions I_(P4) in thebright image frame and the dark image frame corresponding to the fourthpixels P₄, as shown in FIG. 4. More specifically, the third subframeF_(P3) is formed by gray level data outputted by the third pixels P₃,and the fourth subframe F_(P4) is formed by gray level data outputted bythe fourth pixels P₄.

Referring to FIG. 1 again, the processor 13 includes a light controlmodule 131, a selection module 133, a differential module 135, an offsetcalculation module 137 and an application module 136, wherein the lightcontrol module 131, the selection module 133, the differential module135, the offset calculation module 137 and the application module 136are implemented by software and/or hardware without particularlimitations. For illustration purposes, the light control module 131,the selection module 133, the differential module 135, the offsetcalculation module 137 and the application module 136 are shown to beseparated from each other herein, but operations thereof are consideredto be accomplished by the processor 13. The processor 13 preferablyfurther includes a storage element 130 configured to previously storeparameters required during operation, e.g., a predetermined offset rangecorresponding to an operable range.

The light control module 131 is configured to control the infrared lightsource 15 to turn on or turn off corresponding to the image capturing ofthe image sensor 11 such that the image sensor 11 captures and outputs abright image frame corresponding to the turning-on of the infrared lightsource 15, and captures and outputs a dark image frame corresponding tothe turning-off of the infrared light source 15.

The selection module 133 of the processor 13 is configured to select anoperation mode according to the image frame F outputted by the imagesensor 11. In one embodiment, the selection module 133 of the processor13 selects the first mode or the second mode according to an averagebrightness of the dark image frame. For example, the selection module133 of the processor 13 calculates only an average brightness of theinfrared image regions I_(inf) of the dark image frame or calculates anaverage brightness of the whole dark image frame, and compares theaverage brightness with a brightness threshold (e.g., stored in thestorage element 130). When the average brightness is smaller than thebrightness threshold, it means that the ambient light La is not toostrong and thus the first mode is entered. Accordingly, the first modeis referred to a normal mode or a weak light mode herein. When theaverage brightness is larger than the brightness threshold, it meansthat the ambient light La is strong and thus the second mode is entered.Accordingly, the second mode is referred to a strong light mode herein.

As mentioned above, when the ambient light La is too strong, thebrightness difference between the bright image frame and the dark imageframe is not obvious. Accordingly, in another embodiment, the selectionmodule 133 of the processor 13 selects the first mode or the second modeaccording to an average brightness difference between the bright imageframe and the dark image frame. When the average brightness differenceis larger than a brightness difference threshold (e.g., stored in thestorage element 130), it means that the ambient light La is not toostrong and thus the first mode is entered. When the average brightnessdifference is smaller than the brightness difference threshold, it meansthat the ambient light La is strong and thus the second mode is entered.

Referring to FIG. 5, it is an operating method of a first mode of thepresent disclosure. In the first mode, the differential module 135 ofthe processor 13 calculates a difference between the infrared subframeF_(inf_B) of the bright image frame outputted by the image sensor 11 andthe infrared subframe F_(inf_D) of the dark image frame outputted by theimage sensor 11. For example, it is assumed that the infrared subframeF_(inf_B) includes an object image I₉ and a background image Ia, whereasthe infrared subframe F_(inf_D) includes only a background image Ia.When the infrared subframe F_(inf_D) is subtracted from the infraredsubframe F_(inf_B), a differential image (F_(inf_B)−F_(inf_D)) onlycontains the object image I₉ thereby removing the interference from thebackground image.

Referring to FIGS. 1 and 4, the operation of the second mode isillustrated hereinafter. In this embodiment, a dot object 9 located at alight incident side of the condenser lens 10 is taken as an example forillustration purposes. The image sensor 11 captures, based on apredetermined focal length, and outputs an image frame F (e.g., a brightimage frame or a dark image frame) to the processor 13. It is assumedthat the selection module 133 selects to enter the second mode, and theoffset calculation module 137 of the processor 13 divides the imageframe F into a first subframe F_(P1) and a second subframe F_(P2),wherein the first subframe F_(P1) is associated with the first pixels P₁and the second subframe F_(P2) is associated with the second pixels P₂.As mentioned above, when the object 9 is at the second focal length(i.e. the predetermined focal length) of the condenser lens 10, imageregions associated with the object 9 in the first subframe F_(P1) andthe second subframe F_(P2) are substantially at corresponding positionswithout having an offset. When the object 9 is not at the second focallength of the condenser lens 10, image regions associated with theobject 9 in the first subframe F_(P1) and the second subframe F_(P2)have an offset therebetween and are not at corresponding positions. Theoffset calculation module 137 of the processor 13 separates at least oneforeground image according to the first subframe F_(P1) and the secondsubframe F_(P2) of the bright image frame or the dark image frame. Asmentioned above, when the ambient light La is strong enough, thebrightness difference between the bright image frame and the dark imageframe is not obvious, and thus a foreground image is separable accordingto each of the bright image frame and the dark image frame.

For example, FIG. 4 shows that a first image region I₉₁ in the firstsubframe F_(P1) shifts upward by S₁ from a center line (e.g., dashedline), and a second image region I₉₂ in the second subframe F_(P2)shifts downward by S₂ from a center line (e.g., dashed line). The offsetcalculation module 137 of the processor 13 then calculates a firstoffset between S₁ and S₂, e.g., (S₁−S₂). It should be mentioned that thecalculation of the offset is not limited to take the center line as areference line, and the center line is taken herein only forillustration purposes. It is also possible to implement the calculationof the offset by the block matching or motion detection withoutparticular limitations as long as the first offset between correspondedimage regions (e.g., I₉₁ and I₉₂) in the first subframe F_(P1) and thesecond subframe F_(P2) is obtainable, wherein whether two image regionsare corresponded to each other is identified according to, for example,the brightness or shape of the two image regions. The offset calculationmodule 137 of the processor 13 identifies at least one image regioncorresponding to the first offset within a predetermined range as atleast one foreground image, and identifies the image region outside thepredetermined range as the background image.

When the pixel matrix 111 includes the arrangement of four pixel types,the offset calculation module 137 of the processor 13 further dividesthe image frame F into a third subframe F_(P3) and a fourth image frameF_(P4), wherein the third frame F_(P3) is associated with the thirdpixels P₃ and the fourth subframe F_(P4) is associated with the fourthpixels P₄. In the second mode, the offset calculation module 137 of theprocessor 13 separates at least one foreground image according to thethird subframe F_(P3) and the fourth subframe F_(P4) of the bright imageframe or the dark image frame.

For example, FIG. 4 shows that a third image region I₉₃ in the thirdsubframe F_(P3) shifts rightward by S₃ from a center line (e.g., dashedline), and a fourth image region I₉₄ in the fourth subframe F_(P4)shifts leftward by S₄ from a center line (e.g., dashed line). The offsetcalculation module 137 of the processor 13 then calculates a secondoffset between S₃ and S₄, e.g., (S₃−S₄). As mentioned above, thecalculation of the offset is not limited to use the subtraction. Theoffset calculation module 137 of the processor 13 identifies at leastone image region corresponding to the second offset within apredetermined range as at least one foreground image, and identifies theimage region outside the predetermined range as the background image.

As mentioned above, the predetermined range is previously stored in thestorage element 130, and is an offset corresponding to an operable rangeof the imaging device 1. In other words, when the first offset and/orthe second offset exceeds the predetermined range, it means that theimage region belongs to the background image.

It should be mentioned that although FIG. 4 shows that the first imageregion I₉₁ shifts upward by S₁, the second image region I₉₂ shiftsdownward by S₂, the third image region I₉₃ shifts rightward by S₃ andthe fourth image region I₉₄ shifts leftward by S₄, it is only intendedto illustrate but not to limit the present disclosure. The shiftdirection of the image region corresponding to the object 9 isdetermined according to whether the object 9 is close to or away fromthe condenser lens 10 from the second focal length as well as the coverpattern of the opaque layer 113 upon the first pixels P₁ to the fourthpixels P₄, and is not limited to that shown in FIG. 4.

The application module 139 outputs a control signal Sc according to atleast one foreground image bring identified, e.g., outputting thecontrol signal Sc according to a moving direction, speed and a variationof number of the least one foreground image to control the operation ofan application program.

It should be mentioned that, in this embodiment the image region isillustrated with a circle (i.e. corresponding to the dot object 9), butthe present disclosure is not limited thereto. The image region may beany part in the image frame F, e.g., edges, capable of clearly showingthe offset without particular limitations.

In addition, in order to improve the identification accuracy, theprocessor 13 further calibrates brightness of the first subframe F_(P1)and the second subframe F_(P2) to be substantially identical using ashading technique. Accordingly, it is able to correctly identify thecorresponded image regions (e.g., image regions having identicalbrightness) in the first subframe F_(P1) and the second subframe F_(P2),e.g., I₉₁ and I₉₂. When the pixel matrix 111 includes the arrangement offour pixel types, the processor 13 further calibrates brightness of thethird subframe F_(P3) and the fourth subframe F_(P4) to be substantiallyidentical using the shading technique. Accordingly, it is able tocorrectly identify the corresponded image regions (e.g., image regionshaving identical brightness) in the third subframe F_(P3) and the fourthsubframe F_(P4), e.g., I₉₃ and I₉₄.

Referring to FIGS. 1, 2A-2B and 4-6, FIG. 6 is a flow chart of anoperating method of an imaging device according to one embodiment of thepresent disclosure. The operating method is, for example, applicable tothe imaging device 1 of FIG. 1. As mentioned above, the imaging device 1includes an infrared light source 15, a plurality of infrared pixelsP_(inf), a plurality of first pixels P₁, a plurality of second pixels P₂and a plurality of microlenses 115. The first pixels P₁ and the secondpixels P₂ respectively receive incident light of different phasesthrough a first part and a second part of the microlenses 115. Forexample, FIG. 1 shows that the first part is at a lower part of thepixel and the second part is at an upper part of the pixel, but theposition and ratio with respect to the pixel of the first and secondparts are not limited to those shown in FIG. 1.

The operating method of this embodiment includes the steps of:outputting, by an imaging device, a bright image frame when an infraredlight source is turned on and a dark image frame when the infrared lightsource is turned off (Step S61); respectively forming an infraredsubframe, a first subframe and a second subframe according to infraredimage regions corresponding to infrared pixels, according to first imageregions corresponding to first pixels and according to second imageregions corresponding to second pixels (Step S62); selecting a firstmode or a second mode (Step S63); calculating, in the first mode, adifferential image between the infrared subframe of the bright imageframe and the infrared subframe of the dark image frame to separate atleast one foreground image (Step S64); and separating, in the secondmode, at least one foreground image according to the first subframe andthe second subframe (Step S65).

Step S61: The light control module 131 of the processor 13 controls theinfrared light source 15 to turn on and turn off corresponding to theimage capturing of the image sensor 11 such that the image sensor 11outputs a bright image frame corresponding to the turning-on of theinfrared light source 15 and outputs a dark image frame corresponding tothe turning-off of the infrared light source 15. More specifically, aframe rate of the image sensor 11 is at least twice of a lightingfrequency of the infrared light source 15, e.g., two times, four timesand so on.

Step S62: The image sensor 11 outputs every image frame F being captured(e.g., the bright image frame and the dark image frame) to the processor11 for the post-processing. For example, the processor 11 forms aninfrared subframe F_(inf) according to infrared image regions I_(inf) inevery image frame F corresponding to the infrared pixels P_(inf), formsa first subframe F_(P1) according to first image regions I_(P1) in everyimage frame F corresponding to the first pixels P₁, and forms a secondsubframe F_(P2) according to second image regions I_(P2) in every imageframe F corresponding to the second pixels P₂, as shown in FIG. 4. Themethod of forming the subframe is to recombine pixels according to theiroriginal positions in the image frame F so as to form the infraredsubframe F_(inf), the first subframe F_(P1) and the second subframeF_(P2).

In some embodiment, the imaging device 1 further includes a plurality ofthird pixels P₃ and a plurality of fourth pixels P₄ respectively receiveincident light of different phases through a third part and a fourthpart of the microlenses 115. For example, FIG. 1 shows that the thirdpart is at the right side of the pixel and the fourth part is at theleft part of the pixel, but the position and ratio with respect to thepixel of the third part and fourth parts are not limited to those shownin FIG. 1. In the embodiment shown in FIG. 1, the first part and thesecond part are at two opposite sides along a first axis (e.g., X axis)of the microlenses 115, and the third part and the fourth part are attwo opposite sides along a second axis (e.g. Y axis) of the microlenses115.

When the pixel matrix 111 includes the arrangement of four pixel types,in the Step S62 the processor 11 further forms a third subframe F_(P3)according to third image regions I_(P3) in every image frame Fcorresponding to the third pixels P₃, and forms a fourth subframe F_(P4)according to fourth image regions I_(P4) in every image frame Fcorresponding to the fourth pixels P₄. In some embodiments, in the firstmode the differential module 135 of the processor 13 generates theinfrared subframe F_(inf) without generating the first subframe F_(P1)to the fourth subframe F_(P4); and in the second mode, the offsetcalculation module 137 of the processor 13 generates the first subframeF_(P1) to the fourth subframe F_(P4) without generating the infraredsubframe F_(inf).

Step S63: The selection module 133 of the processor 13 identifieswhether ambient light is too strong according to average brightness ofthe image frame F. In one embodiment, the selection module 133 of theprocessor 13 selects a first mode or a second mode according to anaverage brightness value of the dark image frame. In another embodiment,the selection module 133 of the processor 13 selects a first mode or asecond mode according to an average brightness difference between thebright image frame and the dark image frame. In the present disclosure,the first mode is, for example, a normal mode or a weak light mode, andthe second mode is, for example, a strong light mode. Accordingly, theprocessor 13 is able to select a proper algorithm according to theintensity of ambient light so as to separate at least one foregroundimage.

Step S64: When the ambient light is not too strong, a first mode isentered. In this case, the differential module 135 of the processor 13directly calculates a differential image between the infrared subframeF_(inf_B) of the bright image frame and the infrared subframe F_(inf_D)of the dark image frame to separate at least one foreground image I₉ asshown in FIG. 5. More specifically, gray values outputted by theinfrared pixels P_(inf) are calculated in the normal mode without beingcalculated in the strong light mode.

Step S65: When the ambient light is strong, the second mode is entered.In this case, the offset calculation module 137 of the processor 13calculates at least one first offset (e.g., S₁ and S₂ shown in FIG. 4)according to the first subframe F_(P1) and the second subframe F_(P2),and identifies at least one image region corresponding to the firstoffset within a predetermined range as at least one foreground image. Asmentioned above, the predetermined range is a predetermined offset rangecorresponding to an operable range.

When the matrix pixel 111 includes the arrangement of four pixel types,in the Step S65, the offset calculation module 137 of the processor 13further calculates at least one second offset (e.g., S₃ and S₄ shown inFIG. 4) according to the third subframe F_(P3) and the fourth subframeF_(P4), and identifies at least one image region corresponding to thesecond offset within a predetermined range as at least one foregroundimage. It should be mentioned that the foreground images identifiedaccording to the first offset and the second offset are not necessary tobe identical, and the identified foreground images may partiallyidentical or totally different according to image features in the imageframe F.

Finally, the application module 139 of the processor 13 outputs acontrol signal Sc according to the identified foreground image toperform different applications, e.g., performing the gesturerecognition.

In addition, in order to improve the identification accuracy, theprocessor 13 calibrates brightness of the first subframe F_(P1) and thesecond subframe F_(P2) to be substantially identical using a shadingtechnique, and calibrates brightness of the third subframe F_(P3) andthe fourth subframe F_(P4) to be substantially identical using theshading technique such that, during calculating the offset, correspondedimage regions in the first subframe F_(P1) and the second subframeF_(P2) are easier to be found, and corresponded image regions in thethird subframe F_(P3) and the fourth subframe F_(P4) are easier to befound.

As mentioned above, the conventional gesture recognition system may notoperate accurately under strong ambient light. Therefore, the presentdisclosure provides an imaging device (e.g., FIG. 1) and an operatingmethod thereof (e.g., FIG. 6) that separate the foreground image usingdifferent algorithms under different ambient light intensity so as toovercome the problems of the conventional gesture recognition system.

Although the disclosure has been explained in relation to its preferredembodiment, it is not used to limit the disclosure. It is to beunderstood that many other possible modifications and variations can bemade by those skilled in the art without departing from the spirit andscope of the disclosure as hereinafter claimed.

What is claimed is:
 1. An imaging device, comprising: a condenser lens;and an image sensor configured to sense light penetrating the condenserlens, the image sensor comprising: a pixel matrix comprising a pluralityof infrared pixels, a plurality of first pixels and a plurality ofsecond pixels; an opaque layer covering upon a first region, which is apart of each first pixel, and upon a second region, which is a part ofeach second pixel, wherein the first region and the second region aresymmetrically arranged in a first direction, and uncovered regions ofthe first pixels and the second pixels are arranged to be larger at apixel edge than at a pixel center; a plurality of microlenses; and aninfrared filter layer covering upon the infrared pixels.
 2. The imagingdevice as claimed in claim 1, wherein the pixel matrix further comprisesa plurality of third pixels and a plurality of fourth pixels, the opaquelayer further covers upon a third region, which is a part of each thirdpixel, and upon a fourth region, which is a part of each fourth pixel,and the third region and the fourth region are symmetrically arranged ina second direction.
 3. The imaging device as claimed in claim 2, whereinthe first region, the second region, the third region and the fourthregion are 5% to 95% of an area of a single pixel.
 4. The imaging deviceas claimed in claim 2, wherein the third pixels and the fourth pixelsare arranged at a part of pixel rows of the pixel matrix.
 5. The imagingdevice as claimed in claim 1, wherein one first pixel having the firstregion covered by the opaque layer and one second pixel having thesecond region covered by the opaque layer are two adjacent pixels in thefirst direction.
 6. The imaging device as claimed in claim 1, whereinthe uncovered regions of the first pixels and the second pixels arearranged to increase or monotonically increase toward the pixel edge. 7.The imaging device as claimed in claim 1, wherein the first pixels andthe second pixels are arranged at a part of pixel columns of the pixelmatrix.
 8. The imaging device as claimed in claim 1, wherein theinfrared pixels are not covered by the opaque layer.
 9. The imagingdevice as claimed in claim 1, wherein the opaque layer and the infraredfilter layer are sandwiched between the pixel matrix and themicrolenses.
 10. The imaging device as claimed in claim 1, wherein adistance between the opaque layer and the pixel matrix is identical tothat between the infrared filter layer and the pixel matrix.
 11. Animaging device, comprising: a condenser lens; and an image sensorconfigured to sense light penetrating the condenser lens, the imagesensor comprising: a pixel matrix comprising a plurality of infraredpixels, as well as a plurality of first pixels and second pixelsarranged at a part of pixel columns of the pixel matrix; an opaque layercovering upon a first region, which is smaller than an area of coveredfirst pixel, of each first pixel, and upon a second region, which issmaller than an area of covered second pixel, of each second pixel,wherein uncovered regions of one first pixel and one second pixel aresymmetrically arranged and adjacent to each other; a plurality ofmicrolenses; and an infrared filter layer covering upon the infraredpixels.
 12. The imaging device as claimed in claim 1, wherein the pixelmatrix further comprises a plurality of third pixels and fourth pixelsarranged at a part of pixel rows of the pixel matrix, the opaque layerfurther covers upon a third region, which is smaller than an area ofcovered third pixel, of each third pixel, and upon a fourth region,which is smaller than an area of covered fourth pixel, of each fourthpixel, and uncovered regions of one third pixel and one fourth pixel aresymmetrically arranged and adjacent to each other.
 13. The imagingdevice as claimed in claim 12, wherein the first region, the secondregion, the third region and the fourth region are 5% to 95% of an areaof a single pixel.
 14. The imaging device as claimed in claim 11,wherein the uncovered regions of the first pixels and the second pixelsare arranged to increase or monotonically increase from a pixel centertoward a pixel edge.
 15. The imaging device as claimed in claim 11,wherein the infrared pixels are not covered by the opaque layer.
 16. Theimaging device as claimed in claim 11, wherein the opaque layer and theinfrared filter layer are sandwiched between the pixel matrix and themicrolenses.
 17. The imaging device as claimed in claim 11, wherein adistance between the opaque layer and the pixel matrix is identical tothat between the infrared filter layer and the pixel matrix.
 18. Animaging device, comprising: a condenser lens; and an image sensorconfigured to sense light penetrating the condenser lens, the imagesensor comprising: a pixel matrix comprising a plurality of infraredpixels, a plurality of first pixels and a plurality of second pixelsarranged in a first direction and a second direction; an opaque layercovering upon a first region which is a part of the first pixels andupon a second region which is a part of the second pixels, wherein thefirst region and the second region are symmetrically arranged in thefirst direction, and an uncovered region of one first pixel is adjacentto an uncovered region of one second pixel in the first direction; aplurality of microlenses; and an infrared filter layer covering upon theinfrared pixels.
 19. The imaging device as claimed in claim 18, whereinthe pixel matrix further comprises a plurality of third pixels and aplurality of fourth pixels, the opaque layer further covers upon a thirdregion which is a part of the third pixels and upon a fourth regionwhich is a part of the fourth pixels, and the third region and thefourth region are symmetrically arranged in a second direction.
 20. Theimaging device as claimed in claim 18, wherein the first region, thesecond region, the third region and the fourth region are 5% to 95% ofan area of a single pixel.